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ABSTRACT 

This paper presents a model for the evaluation of 
postsecondary online distance education courses and programs. To better 
address the unique nature and audience for these courses and programs, and 
the related institutional needs for assessing their success or failure, the 
focus is on a model from corporate training that provides a comprehensive, 
multi-level evaluation of a distance course or program. This model is based 
on Kirkpatrick’s (1975) levels of evaluation. A general description is 
provided of how this model can be implemented via online means to provide the 
assessment information required by postsecondary institutions. (Contains 12 
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Abstract: In this paper, we present our model for the evaluation of postsecondary online 
distance education courses and programs. To better address the unique nature and 
audience for these courses and programs, and the related institutional needs for assessing 
their success or failure, we focus on a model from corporate training that provides a 
comprehensive, multi-level evaluation of a distance course or program. This model is 
based on Kirkpatrick’s (1975) levels of evaluation. We provide a general description of 
how this model can be implemented via online means to provide the assessment 
information required by postsecondary institutions. 



Background 

Over the past decade, online instruction has been become a popular delivery option in many institutions of higher 
education (Green, 1997). One criticism leveled against conventional “student evaluations of teaching” suggests that 
when used alone, these provide limited information about student comfort without addressing significant factors 
such as student learning and ability to transfer that learning to real-world situations (Trout, 2000). Moreover, many 
evaluation systems for online learning emphasize usability and reliability of the technology over the value of the 
instruction to the students (Anglin & Morrison, 2000). Finally, the unique nature of E-leaming suggests that 
traditional course and program evaluation may be particularly deficient in providing necessary data that can help 
drive further development and support of effective online instruction (Clark, 2000). 

The needs driving the evaluation of E-leaming in postsecondary education coincide with the classical reasons for 
evaluating any type of learning experience as well as expand on them. As with any evaluation of conventional 
instruction, instructors, administrators and other stakeholders in online learning want to know the value of their 
course and program offerings in terms of student satisfaction and student learning. This includes: 

• Student satisfaction with the learning process (interactions with course/program materials, instmctor and 
other students) 

• Appropriate level, pace, depth and breadth of instruction 

• Accomplishment of intended instructional objectives 

• Areas in need of further instructional and pedagogical development 

Although most university evaluation systems do not address transferability of learning to any significant degree, 
since students taking distance education courses are often out in the workplace, and facing real- world demands from 
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clients and supervisors, it is especially important that E-leaming assessment activities address whether the learning 
that takes place is useful to the students. Information collected about this process might include whether: 



• The content of the learning provides a basis of knowledge that can be built upon in later instructional 
situations 

• The learning can be transferred to enhance performance in real-world settings 

In addition to these basic assessment needs for instructors, other stakeholders in the online learning process 
(administrators, technical support, designers, etc.) also hope to measure these facts about the technologies used: 

• Performance of technologies used 

• Usability of the technologies 

• Reliability of the technologies 

• Compatibility of the technologies with each other and existing student technologies 

Because E-leaming is generally a new and unproven method of instructional delivery that faces bias against its 
efficacy, cost efficiency, and quality from those in traditional academia (Ashworth, 1996; Cordes, 1998), it faces 
additional institutional demands for accountability, such as: 

• The costs and benefits of installing and maintaining the technology infrastructure required for distance 
education. 

• Concerns about the effect of distance education efforts to the institutional reputation 

• The need to collect data to assist in future institutional decision-making 

The Significance of Kirkpatrick’s Model to Online Instruction 

In general, a rigorous online evaluation system should measure the success of the content, process, and delivery of 
online instruction in terms of the individual student, the program and/or institution, and/or the field in general. It 
should also be able to address the efficacy of its chosen pedagogy and technology. This paper seeks to adapt a 
model for the evaluation of B-leaming from the field of training and development that takes into account a number 
of different aspects of the online learning process. This four-level model of evaluation, developed by Don 
Kirkpatrick (1975) is currently utilized by 67% of businesses that evaluate their training (ASTD, 1997). However, 
while there have been a few efforts to apply Kirkpatrick’s levels to evaluate distance education in a higher education 
setting (Essex & Cagiltay, 2001; Hack & Tarouco, 2000), this practice is still far from commonplace. 

Once adapted to higher education needs and perspectives, Kirkpatrick’s evaluation model can be used to address the 
many layers of questions that exist about E-leaming. Specifically, Kirkpatrick’s model consists of 4 levels, each one 
more extensive in its purview of instructional effectiveness than the one preceding it and each stage relying on 
achievements gained in the one preceding it for success. Therefore, it is a cumulative model that can be used to 
provide in-depth information about the value of an online course or program. Within each of the four levels 
described below, a systematic method of evaluation is integrated into the process that specifies the goals, objectives, 
criteria, and methods of assessment. 

The Proposed Evaluation Model 
Level 1: Student Satisfaction 
Identify Goals 

As the course was being developed, instructors and developers should have identified the goals associated with 
attaining student satisfaction with the course or program in terms of both instmctional design and the delivery 
technology. These goals are fairly generalized, such as making sure the course is attractive and accessible to 
students, and that students feel comfortable and competent with the course requirements. Ideally, this is done as part 
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of the initial course design process, but if these goals were not previously explicitly identified, the evaluation team 
will need to work with the course designers to extract them before the evaluation process begins. 

identify Objectives 

Objectives need to be based on the overall goals of the course, which are transformed into objectives through 
specific and concrete definition. The developers of an online course evaluation should measure student satisfaction 
based on the objectives regarding the instructional materials, the instructional strategies, and the types of interactions 
provided by the course. In terms of delivery technology, evaluation measures should look at specific objectives in 
terms of the ability of the hardware, software, and networking to meet student expectations and needs. Again, if 
these objectives were not previously explicitly identified, the evaluation team will need to work with the course 
designers to extract them before the evaluation process begins. 

Develop Performance Criteria 

In the design phase, developers and instructors of online courses need to determine appropriate criteria for 
measuring the success (student satisfaction) of the instruction, and these criteria need to be based on the course goals 
and objectives previously decided upon. Furthermore, Clark (2000) points out that it is important to separate 
instructional technology from delivery technology when evaluating online courses. Some examples of performance 
criteria for instructional technology are: whether the course met student goals; if the level of instruction was 
appropriate to the student audience; if the pace was appropriate; if the course covered an appropriate range and 
depth of content; and if the student-student and student -instructor interactions in the course were considered to be 
helpful by students to the learning process. Criteria for delivery technology might include whether it was 
compatible with student equipment and existing institutional systems, and its reliability, usability, and speed. 

Specify Data Collection Method 

Evaluation at Kirkpatrick Level 1 can be done on either a summative (end-of-term) or formative (midterm) basis, or 
both The most common method of collecting Level 1 data is through a survey instrument, usually administered as a 
summative student course evaluation that asks student opinions on whether the course fulfilled their objectives. 
Document analysis is also a popular method of evaluation with online courses, since the unique nature of an E- 
course is that a record of all discussions, assignments and group work is maintained. Other statistics, such aswhen, 
how often, and how long students access various aspects of the course, are also available in server log documents. 
There are also options to collect formative data that can be used during the course of the instruction to guide changes 
in course delivery and design. Focus groups provide students a forum in which to build on each other’s impressions 
of the course, and these can be implemented in secure online forums, where students can provide open and honest 
feedback that is later summarized anonymously for the instructor.. Individual student interviews, while more time - 
consuming, can allow for deeper expbration of student attitudes and experiences with the course. Observations of 
the course by a faculty colleague or instructional consultant (attained by allowing them access to the course 
discussion forums and other documents and communication) can also reveal much about student engagement, levels 
of interaction, and general conduct of the course. 

Level 2: Student Learning 

Identify Goals of the Instructional Effort 

Cross & Angelo (1993) identify five basic areas of higher education from which instructional goals are drawn. They 
are: higher order thinking skills (problem solving, analysis, synthesis);basic success skills (reading, writing, 
listening, speaking); field dependent knowledge and skills (methods, materials, tools); academic perspectives and 
values (objectivity, rigor, collaboration);and personal development attitudes and skills (responsibility, leadership, 
commitment). Theoretically, the student learning goals for an online course do not differ from those of a traditional 
course. However, competency with the technology itself always, whether implicitly or explicitly, comprises part of 
the online instructional process and the basic requirements for success in an online course. 



Identify Objectives of the Instructional Effort 



Some objectives related to higher order thinking skills might be: identifying the important elements of a problem; 
understanding the relationship between the elements; and determining what changes in which elements need to be 
made to solve a problem. Basic success skills objectives might include: understanding what is important and what is 
secondary in written information; articulating ideas in written format; listening to others' opinions and ideas; and 
articulating ideas in verbal interactions and presentations. Objectives relating to field-dependent knowledge may 
encompass; understanding which methods are appropriate and their uses in the field; identifying appropriate 
materials and their characteristics; and selecting the best tools for the task(s) and knowing how to utilize them. 
Objectives developed under academic perspectives and values might be: learning how to approach intellectual 
problems with objectivity; understanding the thoroughness required for rigor; describing and defining practices that 
lead to teamwork and accomplishing collaborative goals. Personal development attitudes and skills objectives might 
include: responsibility to one’s work within the discipline; leadership in the field; and commitment to one’s own 
values. 

Develop Performance Criteria 

Defining performance criteria for assessing student learning is normally part of any instructional module or course, 
and leads the evaluation of student learning. Specifically, criteria generally fall into two categories of student 
performance (demonstrating learning): those that pertain to understanding of content; and those that pertain to the 
mastery of a process. 

Specify or Review Existing Assessment Methods 

Except for evaluation of the student’s mastery of technology, this level of evaluation may borrow heavily from 
established course materials and evaluation instruments; e.g., examinations, written assignments, projects, and 
portfolios. These evaluations for the E-course can then be analyzed in exactly the same way as they are in 
traditional, on-campus courses, such as providing average student scores and grade percentages for individual 
assignments to grade point averages for the course. Student mastery with the technology can be tested directly, such 
as by giving them technology-related tasks to perform, or inferred by their success with the assignments in general. 
If grades are not an issue, mastery learning techniques will provide instructors with the number of modules or 
instruments completed at a given proficiency level (which should ideally be at or near 100%). 

Level 3: Transfer of Learning 

Identify Goals of Transfer Learning 

The goals of Level 3 evaluation are twofold. First of all, learning in an academic setting is often considered to be 
sequential and progressive. That is, basic knowledge and understanding must be acquired before students may 
“advance” to higher order thinking and problem solving. This is the philosophy behind offering some courses in 
sequence, and requiring that the preceding course be successfully completed before moving on to the next level. 
Secondly, the goal of much higher education is that student learning will be transferred to reaHife 
situations/problems (such as internships, student teaching, service learning projects, professional tasks, etc.). 
Whether E-leaming focuses on the professional development needs of those already in the field, or are a part of 
university programs in professional education, successful transfer of learning is usually the ultimate goal of the 
educational process. Thus, they often encompass the goals of both pre-service professional training and the further 
development of professionals in the field. Transfer of learning, though, is often difficult to evaluate, since it must be 
evaluated after the learning process takes place, and in a setting other than the real or virtual classroom. Universities 
are not accustomed to tracking student learning post instruction, except for programs with comprehensive or 
qualifying exams or summative portfolios. However, since transfer of learning is so important, any rigorous 
assessment program must include this aspect in its design. 

Identify objectives 
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The objectives of transfer learning are diverse and depend heavily on the field of study, but in general the objectives 
will address students’ ability to access specific skills and relevant knowledge post instruction, and their knowledge 
about when and how to use these skills and relevant knowledge for real-world problems and tasks. 

Develop Performance Criteria 

Specific performance criteria for transfer learning will depend heavily upon the field of study and/or the professional 
field involved, and the level at which the students work in that field. Professional fields often determine the criteria 
that their members must meet through granting licenses and certification. Academic fields typically measure 
performance through instruction, research and publication. Subprofessional performance criteria in both of these 
fields often approximate professional tasks, but at a lower level of mastery. 

Specify Methods to Evaluate Transfer Learning 

In both academic and professional settings, the transferability of learning can be assessed using simulated, authentic 
scenarios or case studies in which the students’ problem-solving skills are put to test. In order to approach real-life 
situations, cases must be complex, ill -defined (Savery & Duffy, 1996) and require the use of higher-order thinking 
skills on the part of the students. To further authenticate the process, when the task would, in the real-world, be a 
^ collaborative effort, students can be required to work in teams so that the skills of communication, negotiation, and 
compromise are also implicated in the solution. In these scenarios or cases, roles can either be initially defined for 
the students or can evolve naturally from the process. Another option for assessing transfer learning, if the students 
are employed or interning in a real-world professional/academic environment, is that the students may be observed 
in reaHife situations by supervisors, by colleagues, and by clients (students). Another option would involve the 
students submitting a video or audiotape of a job-related performance for evaluation. 

Level 4: Return on Investment or Business/Academic Impact 

Identify Goals 

Whereas in a corporate training program the return on investment (ROI) of any online training effort is measured in 
terms of whether or not the cost of the training is more or less than the additional profit gained through performance 
improvement based on the training, academic programs define success in different ways. While the financial aspect 
still exists for the academic world, because most students hope to be in a better financial position after a course of 
study than before, there may be other indicators of success for students that they count into their calculations of ROI, 
such as increased job satisfaction, lower stress, etc. In any case, deciding whether a course “was worth it,” is often a 
very personal kind of ROI calculation, and difficult to quantify. Looking at ROI from an institutional perspective, 
colleges and universities often look to intermediary indices, only some of which indirectly refer to the financial 
health of the organization. Specifically, academic institutions and programs often use academic reputation, program 
rankings, and accreditation status as indicators of success. Although the reputation is somewhat ephemeral, high 
program ranking established by a reputable organization and accreditation through appropriate organizations are 
attributes that institutions of higher education that offer E-leaming are not willing to sacrifice. Other indicators of 
program success are the success of graduates of online programs as well as increasing popularity with the E-leaming 
program for students. 

Identify Objectives 

Calculating objectives related to ROI from an individual student perspective can be, as stated above, a very personal 
process, involving an estimation of how much the learning from a given course or program improves the life of the 
learner in terms of fulfilling monetary, career and personal goals. From the institutional perspective, objectives 
relating to academic or program reputation may also be difficult to decide upon, since much of reputation is 
established through word-of-mouth and on other informal bases. School and/or program rankings, however, 
generally occur at predictable intervals through identifiable evaluative agencies. Therefore, objectives can be set 
which relate to these specific evaluative events. In addit ion, objectives can be set related to a positive change in the 
type and level of positions attained by graduates of the program or institution, and/or positive changes in 
institution/program enrollment rates. 
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Develop Criteria for Positive Changes 

Again, students will determine their own individual criteria for assessing the ROI relative to a given course or 
program of study. The threshold for which the student decides that the course or program was “worth it” may be 
difficult to quantify, but most students will have an implicit sense of it. From the institutional perspective, given the 
imprecise nature of measuring a concept such as institutional reputation, criteria for assessing an institution's or 
program’s reputation may merely focus on the documentation of a stakeholder’s opinions regarding positive 
changes. Positions attained by graduates of a program can be evaluated along the lines of perceived status, entry 
salaries, and/or numbers hired within a certain period of time. Rankings and enrollment rates, however, can reveal 
quantitative changes, in which positive changes can be tracked and documented. Care must be taken to ensure that 
perceived positive change is not the result of other, external factors such as changes in the nature of the field, 
changes in professional demands, changes in the availability of resources, or changes in other policies or regulations. 

Develop Evaluation Methods 

Evaluating student assessment of ROI may simply consist of asking them, in a survey or interview format, whether 
they regard the benefits of the course of study to have been greater than the associated costs. Since the objectives of 
each student and the criteria for measuring these objectives are likely to be so idiosyncratic, it would be difficult or 
peihaps impossible to get at a more objective assessment. From the institutional side, evaluation methods for 
determining the impact of an online program on an institution or program can be accomplished through: inspection 
of published ratings/rankings; the hiring rates of graduates; the level of positions obtained by the graduates; the 
accreditation and licensing status of graduates; the accreditation status of the program, and enrollment rates. These 
measures all provide data that can be used to indicate program success. Data may also be collected on the status of 
graduates through surveys, interviews, and focus groups with them. 



Conclusion 



The major advantage of the Kirkpatrick model of evaluation is that it fosters a thorough assessment of the 
instructional process by viewing the process systematically, through four different lenses. Most college and 
university efforts to evaluate E-leaming are not done systematically and fall short of their goals, either by 
emphasizing student satisfaction for the process over the actual learning achievements or lack thereof, by 
concentrating on positive or negative aspects of the technology while overlooking the learning process altogether, 
and/or by ignoring whether or not the learning will transfer to the real world environments that the learner will face. 
The Kirkpatrick model can also be used to integrate evaluation on the local scale with a more systematic and 
administrative look at institutional and programmatic influences. This new evaluation method helps put into 
perspective data that can inform future university decisions about their technology infrastructure, the professional 
development of faculty involved in online learning, and the prospective role of E-leaming in university programs. 
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